Force from Motion: Decoding Control Force of Activity in a First Person Video

A first person video delivers what the camera wearer experiences while physically interacting with surroundings. In this paper, we focus on a problem of Force from Motion—decoding control force and torque that drive the activity—from a first person video. We use two dominant physical factors encoded in the video to understand the wearer’s activity: physical coordinate system and active force component. (1) A physical coordinate system aligned with gravity in metric scale is a prerequisite for force computation. We learn the gravity cue that exists in a natural image to predict the 3D gravity direction given a sequence of images. The sense of physical scale is revealed to us when the body is in a dynamically balanced state. We compute the unknown physical scale in metric unit by leveraging the torque equilibrium at a banked turn. (2) The active force and torque act on the wearer’s body, reflected in the 3D camera egomotion. We model the dynamics using an inverted pendulum that is simple yet expressive enough to describe force interactions. The optimal active control components that generate the camera motion are computed such that it minimizes 2D reprojection error. As a by-product, we can also recover the passive forces such as air drag, friction, and ground reaction/lifting force. Our method shows quantitatively equivalent reconstruction comparing to IMU measurements in terms of gravity and scale recovery and outperforms the methods based on 2D optical flow for an active action recognition task. We apply our method to first person videos of mountain biking, urban bike racing, skiing, speedflying with parachute, and wingsuit flying where inertial measurements are not accessible.